Business Analyst Community & Resources | Modern Analyst

Business Analysis in the Age of AI

Pulkit Singhal — Fri, 01 May 2026 19:24:00 GMT

Business analysis work has become faster and more efficient over the past few years. Requirements are documented more quickly, discussions are summarized sooner, and solution options are produced earlier in the delivery cycle than ever before. Yet many Agile and product teams are discovering an unexpected truth: as delivery accelerates, the importance of human judgment increases rather than diminishes.

The central question facing business analysts today is no longer whether tools and automation belong in analysis work, but where judgment must take precedence. That distinction matters because the most serious failures in delivery rarely come from obvious mistakes. They emerge from reasonable decisions that appear correct at the time and gradually move teams off course.

Where Acceleration Helps and Where It Falls Short

Modern analysis practices are excellent at speeding up work that is inherently mechanical:

Converting discussions into draft requirements
Identifying patterns across large volumes of data
Refining user story language
Summarizing customer or stakeholder feedback

When used well, this removes low‑value effort from the analyst’s workload. When relied upon uncritically, it creates the illusion of progress.

The challenge is not poor quality output. The real risk lies in outputs that are clear, structured, and confident enough to pass surface review, while subtly reinforcing incorrect assumptions. This is where judgment becomes decisive.

Judgment Gap #1: Determining Whether a Requirement Is Worth Building

Clear and complete requirements do not guarantee meaningful outcomes.

In day‑to‑day delivery, analysts encounter familiar patterns:

A requirement addresses a visible symptom rather than the underlying problem
Stakeholders agree on wording but diverge on expected results
A feature meets acceptance criteria yet produces no behavioral change

Experienced analysts pause to ask questions that artifacts alone cannot answer:

What decision or behavior is supposed to change as a result of this work?
If this is delivered perfectly and nothing improves, what are we missing?

Strong analysis is not just about expressing requirements well, but about challenging their intent.

Judgment Gap #2: Interpreting Context That Never Appears in Documentation

Business environments contain layers of context that rarely make it into requirements or datasets:

Organizational dynamics and power structures
Regulatory concerns driving risk‑averse behavior
Legacy failures that shape stakeholder trust
Competing incentives across teams

Analysts recognize these signals not because they are documented, but because they have seen the downstream effects:

Solutions that are functionally correct but poorly adopted
Processes that are bypassed in practice
Reports and dashboards that exist but are ignored

Judgment here is not guesswork. It is pattern recognition developed through exposure to real consequences.

Judgment Gap #3: Recognizing When Clarity Creates False Confidence

Early clarity is often welcomed as momentum. Detailed backlogs, well‑defined flows, and polished models can make teams feel aligned and confident.

Seasoned analysts remain cautious.

They ask whether clarity is reducing uncertainty—or simply hiding it:

Are assumptions being locked in too early?
What would invalidate this design once it is tested?
Are open questions being resolved, or quietly deferred?

Sometimes the most responsible decision is to leave things deliberately unresolved, even when tools and processes encourage premature finalization.

What This Means for Business Analysts

As delivery mechanics become faster, the value of business analysis shifts away from producing artifacts and toward exercising judgment:

Framing the right problems
Interpreting conflicting signals
Evaluating consequences under uncertainty
Challenging assumptions before they harden

These capabilities are not procedural skills. They are developed through experience, reflection, and exposure to real outcomes especially failure.

Closing Thoughts

Modern tools and practices have made business analysis more efficient, but efficiency does not replace responsibility. The most effective analysts are not those who produce the most artifacts in the shortest time. They are the ones who know when clarity is helpful, when it is premature, and when the best contribution is to pause and ask a different question altogether.

That work remains deeply human and central to successful delivery.

Harnessing Quantum Computing for Business Analysis

Tosin Clement — Mon, 19 May 2025 20:17:00 GMT

Introduction

Quantum computing has transitioned from a theoretical concept in academia to a technology on the verge of becoming real, with real potential for multiple industries. Therefore, understanding quantum computing will be increasingly important for business analysts, as it will be the most significant development of how we analyze, optimize, and make decisions with data.

Understanding Quantum Computing

Unlike a classical computer, which uses bits (0s, 1s) to capture information, or data, as input, quantum computers use quantum bits, or qubits. Because a qubit can represent 0, 1, or both 0 and 1, based on the principles of superposition and entanglement, a quantum computer can be in a superposition of more than two states. This aspect of quantum computing allows solutions to complex calculations and algorithms to emerge quicker than a classical computer by presenting some problems and algorithms that classical computers could not even begin to solve.

Opportunities for Business Analysts

1. Advanced Data Analysis and Optimization

Quantum computing capabilities will result in the ability to address complex optimization problems that a classical computer would need significantly more time to solve. The introduction of quantum computing to business analytics, optimization to supply chain and routes of products, as well as inventory levels, takes optimization to a new potential plateau of existence. With even more complexity and challenges arriving with introducing new products or services, there will be opportunities to create short-term compliance and solutions to reduce costs and provide increasingly efficient solutions. As developments in quantum computing are made, more applications will continue to emerge. Companies are emerging as leaders sooner in this area, e.g., D-Wave. There are several promising applications and the beginnings of some early demonstrations of quantum solutions in workforce scheduling problems and logistics-based optimization problems.

2. Enhanced Risk Assessment and Financial Modeling

In the Finance arena, the eventual development of quantum computing will allow better risk assessment and risk assessment models with large datasets and complex interactions. JPMorgan and Amazon are developing quantum algorithms for portfolio optimization and risk analysis.

3. Accelerated Drug Discovery and Healthcare Analysis

Speeding up drug discovery and analyzing health care Quantum computing can speed up drug discovery by simulating the interactions of molecules. By accurately modeling complex biological systems, pharmaceutical companies can find potential drug candidates faster, shortening their development time and limiting costs.

4. Improved Machine Learning and AI Integration

Superior machine learning and AI Quantum computing can enhance efficiency for machine-learning algorithms, such as processing massive datasets. This leads to predictive models that are more accurate and have shorter training times. This could have applications in almost every industry, including marketing, healthcare, finance, etc.

Challenges in Adopting Quantum Computing

1. Technical Limitations and Error Rates

The current state of quantum computing relies on Noisy Intermediate-Scale Quantum (NISQ) devices, which present challenges regarding implementation reliability, such as the limited number of constrained qubits and particularly high error rates in executing complex algorithms.

2. High Costs and Infrastructure Requirements

Quantum computers have specialized needs from a safe environment, mostly low-temperature architectures, isolation from any kind of interference, etc. The cost of creating and maintaining the support hardware is substantial, restricting access to quantum technologies for most organizations.

3. Talent Shortage and Skill Gaps

There is a significant shortage of qualified resources regarding quantum computing. Business analysts must learn quantum algorithms and principles regarding how to apply quantum technology. Learning these will likely not involve a short learning window for many, with a significant investment of time, education, and training needed.

4. Security Concerns

Quantum computing poses a risk to present-day encryption. Current quantum computers can not even *currently* break existing cryptography methods; however, if their processing power increases significantly, they WILL break existing cryptographic protocols. Essentially, quantum-resistant encryption must be developed to anticipate quantum computing capabilities.

Preparing for the Quantum Future

To stay ahead, business analysts should:

Learning the principles of quantum computing and its applications
Relate and network with quantum computing scientists and engineers
Design and develop pilot projects to assess potential use cases
Keep up to date with innovations and technological advances..

Conclusion

Quantum computing is likely the next evolution of business analytics purposes and assisting efficiencies in data processing, optimization, and predictive modelling. While these new technologies cause concern for organizations, the effect is often exaggerated. Learning proactively will allow business analysts to promote and apply quantum computing technologies for organizational advantages.

Sankey Diagrams vs Parallel Sets

Charles Friedo — Tue, 02 Jul 2024 08:26:00 GMT

For many years now, a lot of people have found it difficult to identify the difference between Sankey diagrams and parallel sets. The two have made headlines, given that most people find it challenging to note what makes them different from each other. What remains to be undeniable is the fact that the Sankey diagram is among the top data visualization tools mostly used by business owners and data analysts.

A Sankey diagram is mostly used by business owners to evaluate the flow of various data elements within their business environment. Even though there are multiple tools used in data visualization, most of the tools available on the web have multiple names. This is one of the elements that confuse most people who are on the lookout to find out what makes these two charts unique.

Given that there is a lot of confusion, it is high time to address what makes these two charts unique and how they operate. Remember that when investing in data visualization tools, applying the wrong tool is likely to cost you at the end of the day. The reality remains that what makes people confused between a Sankey diagram and parallel sets is their appearance.

Sankey Diagram

The Sankey diagram was invented by an Irish engineer known as Matthew Sankey, who developed it with the aim of evaluating the energy efficiency of a steam engine. Besides, the Sankey diagram is considered to be the most complex type of data visualization, considering how it processes data and the context of its output. The chart outlines the flow of data from one point to another within a company environment.

A Sankey diagram uses different flow paths that come with varying sizes, depending on the volume of the flow. It elaborates on how data enters the system and leaves to showcase the final product and how all the activities take place. The path on the chart is used to showcase the direction of data flow across the entire system. What people need to understand is that not all Sankey diagrams use arrows to show the direction of the flow.

This type of Sankey chart shows the direction in a visual manner which makes it easier to communicate the data input and output. The flow of data in a Sankey diagram can be split or combined at any point to showcase how quantities within a system can split after a certain change takes place. In most cases, a color or divider is used to split the Sankey diagram into different categories depending on the nature of the data flow.

The thickness of a particular data flow is directly proportional to the magnitude of the flow. This mode of data visualization enables you to compare and contrast different data quantities that are flowing within an organization or business. Considering this mode of operation, the Sankey chart is the most ideal data visualization tool that can be used to display communication inside an abstract system.

Anytime you want to analyze the direction of your data flow, the Sankey chart gives you a chance to evaluate your data from different angles and assess how it flows from one point to the other, as well as the change that takes place. The chart can also point out the dominant contributors to the entire flow across the system. With a Sankey diagram, you can evaluate cases of inefficiencies within the data visualization system and find out the most ideal ways to solve the problem.

The color combinations used in the Sankey diagram make it an attractive mode of data visualization that your target audience will enjoy viewing. Since the chart displays data at different levels, analysts find it easier to analyze data from different perspectives before making conclusions. If you choose this mode of data visualization, you can acquire a Sankey diagram maker that is designed to do the job on your behalf.

Parallel Sets

Parallel sets were invented in 2005 by three scientists who wanted to use it to analyze their categorical data in a visual way. This mode of data visualization was mainly designed to analyze and extract insights from large volumes of data within minutes. It also has features that can visualize complex data sets and convert them into a simple language that a non-technical audience can understand without interpretation.

This mode of data visualization was mainly propelled by the parallel coordinate plot. After the tool was established, several changes were made on it to make it more advanced and accommodate large volumes of data. Parallel sets offer clear data output that data analysts can enjoy using. The tool contains parallel sets of lines that are outlined depending on the nature of the data under visualization.

On the chart, every set of data is represented with a coloured band, also known as a flow path. The lines are divided into nodes that reciprocate the number of data sets represented. The length of every line slit is mainly determined by the proportional fraction of the categorical data presented. Every line slit on the chart is used to represent a certain part of the whole data relationship.

The lines are outlined in a manner that they maintain the same length across the entire chart. Note that the lines can be divided further into multiple categories, depending on the data you intend to visualize and the nature of the results you intend to generate. Parallel sets are mostly used to showcase the distribution of data within a given system or organization.

Comparing the Differences

Considering the explanation given above, Sankey charts are mainly focused on determining the flow of data between different points. In other words, you can consider a Sankey chart as a simplified flowchart that is more advanced compared to a typical flowchart. What makes a Sankey diagram different from a flowchart is that it can involve different data cycles, which cannot be presented in a flowchart.

The direction of flow in a Sankey chart can split or combine at any point within the system, contrary to the parallel sets. A Sankey diagram is only dedicated to showing the flow of data, and it does not display a series of nodes across the system. When you want to build a Sankey diagram, you only need access to a Sankey diagram maker, which will take responsibility for the remaining task.

Conclusion

When you evaluate the facts outlined in this article clearly, you will realize that there is a major difference between a Sankey chart and parallel sets. Even though the two may appear similar in appearance, they are two different tools that are used to achieve different goals. A Sankey diagram is specifically designed to show the flow of data between two different points or across the entire business environment.

On the other hand, parallel sets are used to show the distribution of data in an organization or business setting. Always keep in mind that the Sankey diagram and parallel sets are completely different aspects, although they are all data visualization tools.

Importance of Data Visualization Tool For Businesses

James Millere — Wed, 07 Jun 2023 05:47:00 GMT

In today's data-driven world, businesses collect and generate vast amounts of data on a daily basis. This data holds valuable insights that can help organizations make informed decisions, identify trends, and drive business growth. However, raw data in its purest form can be overwhelming and difficult to comprehend. This is where data visualization tools come into play. They serve as a bridge between raw data and actionable insights, enabling businesses to effectively analyze and communicate information visually.

Data visualization has become an essential tool for businesses in today's data-driven world. It enables organizations to transform raw data into meaningful insights, empowering them to make informed decisions, uncover patterns, and communicate information effectively. Here are several key points highlighting the importance of data visualization tools for businesses:

1. Simplifying Complex Data:

Data visualization tools help businesses simplify complex datasets by presenting information in a visual format. Charts, graphs, and interactive dashboards make it easier to understand and interpret data, allowing users to quickly identify trends, patterns, and correlations. Visual representations enhance comprehension and enable stakeholders to grasp the significance of the data at a glance.

2. Enhancing Decision-Making:

Effective decision-making relies on accurate and timely information. Data visualization tools provide real-time visualizations of key performance indicators (KPIs), allowing businesses to monitor their progress, identify areas of improvement, and make data-driven decisions. By presenting data in a visually appealing and intuitive manner, these tools facilitate better understanding and enable stakeholders to take decisive action.

3. Facilitating Data Exploration:

Data visualization tools enable users to explore and interact with data, encouraging a deeper understanding of business insights. With the ability to drill down into specific details, manipulate visualizations, and ask questions, users can gain valuable insights and discover hidden trends or outliers. This promotes data exploration and empowers businesses to uncover valuable information that may have otherwise remained unnoticed.

4. Enhancing Communication and Storytelling:

Data visualization is a powerful storytelling tool. Visual representations of data make it easier to communicate complex concepts, trends, and findings to both technical and non-technical stakeholders. By creating engaging visual narratives, businesses can effectively convey information, present compelling arguments, and engage their audience. Visualizations enable stakeholders to grasp information quickly and make informed decisions based on the presented insights.

5. Encouraging Collaboration:

Data visualization tools facilitate collaboration by providing a shared platform for data analysis and interpretation. Team members can access and interact with the same visualizations, promoting a collaborative approach to data-driven decision-making. Visual representations foster a common understanding and enable teams from different departments to align their efforts, share insights, and work towards common goals.

6. Enabling Predictive Analysis:

Data visualization tools can support predictive analysis by displaying historical and real-time data trends. By visually representing data patterns and correlations, businesses can make informed predictions about future outcomes. This empowers organizations to anticipate market changes, identify potential risks or opportunities, and adjust their strategies accordingly.

7. Spotting Anomalies and Outliers:

Data visualization tools help businesses identify anomalies and outliers in their datasets. Visual representations make it easier to detect data points that deviate from the norm, indicating potential issues or exceptional occurrences. By spotting these outliers, businesses can investigate further, take corrective actions, and ensure the accuracy and reliability of their data.

8. Improving Data-Driven Presentations:

Presenting data in a visually appealing manner enhances the effectiveness of data-driven presentations. Data visualization tools allow businesses to create visually compelling and interactive presentations that capture the attention of the audience. By incorporating visualizations into presentations, businesses can convey information more effectively, leave a lasting impression, and facilitate better engagement and understanding.

Conclusion

Data visualization tools are crucial for businesses seeking to leverage the power of data to drive informed decision-making, communicate insights effectively, and gain a competitive edge. These tools simplify complex data, enhance decision-making processes, facilitate data exploration, encourage collaboration, enable predictive analysis, identify anomalies, and improve data-driven presentations. By adopting data visualization tools, businesses can unlock the full potential of their data, gain valuable insights, and thrive in today's data-driven business landscape.

Are you testing your product in the most effective way?

lhytonwatt — Sat, 18 Mar 2023 09:25:00 GMT

Today covid-19 pandemic has completely flipped the business model. Various conventional Business models like banking and health had to adopt the digital era. Even the school and colleges have to create various applications so that they can provide online learning for their students. All these statements justify the modern business model as completely dependent on the performance and the functionality of various web applications. In this condition, the testing of the web application goes a long way in determining a smooth end-user experience. However, due to the rapid increase in the demand for web applications, companies need to improve the efficiency of their testing environment.

In this article, we are going to discuss how Automation Testing can help a company improve the efficiency of the testing environment. We will also discuss various reasons to justify the replacement of manual testing with automation testing in certain areas.

The Creation of Test Automation

During the early years of software development, companies didn’t have any modern technology. So, the developer had to follow the conventional method and test the functioning of the web application with manual intervention. It was a lengthy and time-consuming process as a single human tester had to perform all the testing operations. Companies started to miss their delivery date by huge margins ultimately hampering end-user satisfaction. In this modern Development Industry, you cannot expect a customer to stick to your company just because of brand loyalty. Providing properly functioning high-quality web applications is the only way to maintain user retention. This was one of the primary reasons for the rise of test automation in the Software Development Industry.

Test atomation helped the web application developing companies to save a lot of time by automating the repetitive testing processes in the web application development life cycle. Companies started to witness massive advantages of this extra time. For instance, they can use this time to create innovative features to add in the future update of the web application. However, the company must remember the high initial investment charge that comes under test automation.

Advantages of Test Automation

Many companies hesitate to introduce test automation due to the significantly higher introductory charges. This hesitation is properly visible for individual creators and small startup companies. However, many industry experts suggest that companies should consider test automation as a long-term investment. This point of view will help them to analyze the massive advantages that test automation will provide in the due course of time. To justify this point, let us consider some of the major advantages of automation testing:

Contrary to popular beliefs, Automation Testing is a cost-efficient testing method. In case the developer manages to overlook the initial charges, he will save huge amounts of money in the long run.

Automation testing helps to maintain the proper collaboration between different teams involved in the web application development life cycle. This means that the development, production, testing and quality assessment team can perform synchronized effort creating the applications. Automation Testing also allows various integrations like DevOps to further improve the collaboration between these teams.

Automation Testing plays a major role in improving the quality of the web application. It reduces the work of the developers by performing rigorous regression testing on the web application. This allows the developers to focus on improving the core functionality of the web application.

The massive test coverage and quick test deployment allow developers to quickly complete all the testing processes. This process allows the companies to deliver high-quality applications to their end-users in a short period.

One of the major advantages of automation testing is the deployment of parallel testing. Using this feature, the developers can test the compatibility of the web application on thousands of different browsers, browser versions, devices and OS simultaneously. This process also plays a major role in reducing the overall testing period for the web application.

Automation Testing helps web developing companies to deploy live test coverage. With the help of this feature, the developers do not have to wait for long periods to receive the results. This process also helps them to detect any errors in the core framework of the web application and remove them as soon as possible.

The process to Introduce test Automation in the Most effective way

Importance of the Tool Selection Process

Tools play a major role in improving the efficiency of an automation test environment. However, the developers must remember that they cannot buy any tool from the market and introduce it to the environment. A web developing company must understand the uniqueness of every tool and its specific purpose. The best process of evaluating this uniqueness is to perform market research on the present automation tools. During the tool’s selection, companies should also consider their preferences and project requirements to shortlist the perfect tool according to their needs. It is also a great idea to have proper knowledge about the recent test automation market trends. Let us use LambdaTest as an example to understand the important role of a test automation tool during the web application development life cycle.

LambdaTest is a cloud-based platform to verify the cross-browser compatibility of web applications. Using this platform developers can verify the functionality of the applications using a safe and Secure Selenium grid. LambdaTest also provides end-to-end encryption to securely test the sensitive data involved in various web applications. LambdaTest also helps web developing companies to natively verify the proper functioning of the locally hosted web applications. Due to trustworthiness and unique features, thousands of companies globally use LambdaTest to perform test applications in their web development environment.

The Proper Balance Between Automation Testing and Manual testing

The primary focus of this article was to justify how Automation Testing can replace manual testing in various areas. However, the developers must remember that it is practically impossible to completely replace manual testing with automation testing. This is because certain testing processes are almost impossible to conduct without any form of human intervention. For instance, tests like visual testing and exploratory testing cannot be performed without a manual tester. In visual testing, the developers have to verify the proper placement of all the visual elements like pictures and videos. While on the other hand, exploratory testing requires the knowledge and experience of a human tester. In this form of testing, the developers have to detect all the errors in the web application that were previously overlooked with automation test scripts.

So, the best way to create the most efficient test environment is to maintain an optimal balance between automation testing and manual testing. The developers should also remember that they have to use certain tools that will help to synchronize the test reports from these two sources. Companies should also train those testers more efficiently so that they can implement this efficient form of testing.

Balance Between Native and Cloud Testing

Since the introduction of cloud-based testing, developers have been developing this belief that cloud-based services can completely substitute real devices. However, in reality, virtual machines or emulators can never take the place of real devices. This is because, without device testing, the test results will not be considered conclusive. Now, it is illogical for a small company to spend thousands of dollars to set up a device testing lab. In this situation, the best solution is to use an automation testing tool that does all these things on your behalf.

Currently, there are various tools like LambdaTest that help the developers to test the functionality of the application on not only emulators but also real devices. These test results ensure that none of the physical factors like screen size or device resolution affects the functionality of the application. This is one of the most essential steps to ensure that you provide the most optimal User experience to your customers.

Implementation of Performance Testing

The performance of the web application is one of the most important factors to ensure smooth functioning. While performing performance testing, you must also ensure that none of the external factors negatively affect the application. The external factors can include a bad network connection, the remote location of the customer or an outdated operating system version. All these factors are highly essential to ensure that the web application is accessible to a wider user base. In this competitive web Development Industry, you cannot afford to alienate any segment of the user base just cause of device or platform incompatibility. Proper performance of the web application also helps to control the bounce rate for a web application.

Proper Understanding of the User Base

It is practically impossible for a company to test the functioning of a web application on every platform or web browser. This is because there are thousands of different browsers currently available in the market. Furthermore, more and more new Browsers are added every single day. So, web developing companies must have a proper idea about the preferences of the audience to optimize the test experience. For instance, they must perform user research to understand the most popular device and operating system in the present market. The developers can also analyze the results from Google Analytics to understand this concept. All these factors will help the company to customize the testing experience according to the preferences of the user.

The Final Verdict

So, we can easily conclude that Automation Testing will continue to dominate the web development industry in the coming years. With this form of testing, companies will have immense potential to revolutionize their testing environment. In the coming years, various web developing companies will also integrate advanced Technologies with test automation to improve its efficiency. These Technologies will include cloud-based services and artificial intelligence. Many experts suggest that companies should pay proper attention to training their automation testers. This process will help them to utilize the full potential Automation Testing environment.

The automation testers should also have a basic idea of various programming languages that the company uses. This process will help them to maintain synchronized efforts with the development team and introduce Automation Testing during the early stages of the development cycle. The companies should also remember that the tool selection process has a vital role in ensuring the success of the test environment. These factors will guarantee a bright future for the web developing company in the coming years.

6 Differences Between Data Exploration and Data Presentation

Zach Gemignani — Wed, 16 Mar 2022 18:55:00 GMT

There are big differences between data exploration versus data presentation. And you need to be aware of these differences as you're creating data stories and data presentations. Let’s start by defining our terms:

Data exploration means the deep-dive analysis of data in search of new insights.
Data presentation means the delivery of data insights to an audience in a form that makes clear the implications.

Your toolbox for data exploration tools is flush with technology solutions such as Tableau, PowerBI, Looker, and Qlik. "Visual analytics" tools give analysts a super-powered version of Excel for dicing data to facilitate the search for valuable insights. Flexibility and breadth of features is critical; the user needs to handle lots of data sources and doesn’t know in which direction she will go with the analysis.

Data presentation is a different class of problem with distinct use cases, goals, and audience needs. Think about the incredible data stories delivered by the The Upshot, Fivethirtyeight, and Bloomberg. These data journalists often demonstrate data presentation at its finest, complete with guided storytelling, compelling visuals, and thoughtful text descriptions. When compared to these examples, it becomes obvious that the best efforts by a data exploration tool cannot deliver high-quality data presentation.

You need a specialized solution if you really want to communicate data in ways that engage your audience. To understand the differences between data exploration and data presentation tools, let me offer six key ways that the activities are fundamentally different.

1. Audience — Who is the data for?

For data exploration, the primary audience is the data analyst herself. She is the person who is both manipulating the data and seeing the results. She needs to work with tight feedback cycles of defining hypotheses, analyzing data, and visualizing results.

For data presentation, the audience is a separate group of end-users, not the author of the analysis. These end-users are often non-analytical, they are on the front-lines of business decision-making, and may difficulty connecting the dots between an analysis and the implications for their job.

2. Message — What do you want to say?

Data exploration is about the journey to find a message in your data. The analyst is trying to put together the pieces of a puzzle.

Data presentation is about sharing the solved puzzle with people who can take action on the insights. Authors of data presentations need to guide an audience through the content with a purpose and point of view.

3. Explanation — What does the data mean?

For the analysts using data exploration tools, the meaning of their analysis can be self-evident. A 1% jump in your conversion metric may represent a big change that changes your marketing tactics. The important challenge for the analysts is to answer why is this happening.

Data presentations carry a heavier burden in explaining the results of analysis. When the audience isn’t as familiar with the data, the data presentation author needs to start with more basic descriptions and context. How do we measure the conversion metric? Is a 1% change a big deal or not? What is the business impact of this change?

4. Visualizations — How do I show the data?

The visualizations for data exploration need to be easy to create and may often show multiple dimensions to unearth complex patterns.

For data presentation, it is important that visualizations be simple and intuitive. The audience doesn’t have the patience to decipher the meaning of a chart. I used to love presenting data in treemaps but found that as a visualization it could seldom stand-alone without a two-minute tutorial to teach new users how to read the content.

5. Goal — What should I do about the insights?

The goal of data exploration is often to ask a better question. The process of finding better questions gets to new insights and a better understanding of how your business works.

Data presentations are about guiding decision-makers to make smarter choices. Much of the learning (through data exploration) should be done, leaving the equally difficult task of communicating the insights and the actions that should result.

In all these ways, data exploration and data presentation are different beasts. This is why we’ve chosen to focus on building the best possible data presentation tool, Juicebox.

6. Interactions — How are data insights created and shared?

Data exploration can be a lonely endeavor: Analysts work on their own to gather data, connect data across silos, and dig into the data to find insights. Data exploration is often a solitary activity that only connects with other people when insights are found and need to be shared. That is, when…

Data presentation is a collaborative, social activity. The value emerges when insights found in data are shared with people who understand the context of the business. The dialogue that emerges is the point, not a failure of the analysis.

Finding the Middle Ground: Data Storytelling

There is something between the extreme ends of data exploration and data presentation. We believe data storytelling lies in this intersection. Data stories aren’t entirely about “telling”, nor are they in the wilderness of “finding”. It is the opportunity to explain the data in a guided, narrative way where message meets exploration.

While there are tools for exploration (e.g. Tableau) and tools for presentation (e.g. PowerPoint), it is only recently that you’ve had the change to bring both together in one solution.

Zach Gemignani (zach.gemignani@juiceanalytics.com)

CEO, co-founder, author at Juice Analytics

www.juiceanalytics.com

Cloud Data Warehouse 101

Samuel02 — Thu, 17 Oct 2019 12:00:00 GMT

With the advent of modern-day cloud infrastructure, many business-critical applications like databases, ERPs, Marketing applications have all moved to the cloud. With this, most of the business-critical data now reside in the cloud. Now that all the business data resides on the cloud, companies need a data warehouse that can seamlessly store the data from all the different cloud-based applications. Enter – Cloud Data Warehouse.

This post aims to help you understand what is a cloud data warehouse, its evolution and need. Here are the key things that this post covers:

What is a Data Warehouse?
What is Data Warehousing?
The Early Days of Data Warehousing
Data Warehousing and the Dawn of the Information Superhighway
Data Warehousing and the Advent of Cloud Technology
Benefits of Cloud Data Warehouse
Challenges of Cloud Data Warehouse
Top Five Cloud Data Warehouse Services of 2019

What is a Data Warehouse?

A data warehouse is a repository of the current and historical information that has been collected. The data warehouse is an information system that forms the core of an organisation’s business intelligence infrastructure. It is a Relational Database Management System (RDBMS) that allows for SQ-like queries to be run on the information it contains.

Unlike a database, a data warehouse is optimized to run analytical queries on large data sets. A database is more often used as a transaction processing system. You can read more about the need for a data warehouse here.

Querying the vast data troves present in the warehouse is taxing. This is due to the complex structure of most data warehouse table structures (multiple joins and aggregates) and the sheer amount of data stored. This requires significant computing resources to perform efficiently.

Queries performed on data warehouses allows analysts to glean useful insights into the organisation’s operations. These insights provide guidance to leadership within the company, helping them to make better decisions in improving company performance. This function is best indicated by an alternate name for Data Warehouses: Decision Support Systems.

What is Data Warehousing?

Data warehousing is the combination of various processes and methods used for collecting and storing vast amounts of data for the purpose of query and analysis, in order to generate information and insights for business intelligence.

Getting the data from the business transaction systems to the analytical systems (known as data migration) involves the ETL Process.

The ETL process is used to Extract data from the source systems, Transform the data into a usable, queryable form and then Load said data to the destination database: the data warehouse. This may also involve extracting and combining different data sets from a variety of disparate sources into a singular cohesive form. This process is referred to as data integration.

Before we dive into understanding what is a Cloud Data Warehouse, it is important to understand the history and origin of Data Warehouses.

The Early Days of Data Warehousing

Business Intelligence has been around since analysts realised the benefits of using an organisation’s historical data as a research asset. From the 1960s new methods for managing and analysing vast amounts of data were continuously being developed. As computing systems became more affordable and more powerful, and the amount of data generated grew exponentially, data warehousing would evolve as a tool of business intelligence.

For over half a century the discipline of data science and business intelligence grew and matured into its own discipline and industry. And, as the methods and paradigms improved, so did the technology that would form the infrastructure for data warehousing.

While the concept of data warehousing was initially provided by American computer scientist, Bill Inmon, it can be said that Data Warehousing officially began in the late 1980s with the formation of the Business Data Warehouse. At this time the internet was a large, private computer network that, while spanning the continental United States, was only accessible by government and military organisations, renowned academic institutions and large corporations.

These entities would communicate via dedicated phone lines over the existing telecom infrastructure, migrating data to expensive onsite data warehouse servers. During this period bandwidth was extremely expensive and had to be carefully managed. This led to the practice of migrating data during non-work hours, otherwise known as the batch window. However, the internet was soon to “go public”, and that shift would lead to significant improvements.

Data Warehousing and the Dawn of the Information Superhighway

The internet began as a military project developed to enable persistent communication between diverse military divisions and the military’s Central Command and Control. However, upon the inclusion of academic institutions and large corporations, it was evident that it had potential far beyond its initial military applications.

When the internet became accessible to the public in the early-to-mid 90’s it led to a surge in the expansion and evolution of its infrastructure. The increased demand meant that bandwidth would become cheaper and vastly improve in speed and capacity. As a result of this, organisations that performed data migrations were no longer restricted to run the ETL process during the batch window and so systems could be regularly updated throughout the day.

Several new data integration and migration processes were developed to take advantage of the increased capacity, such as Message-Oriented Movement and Data Replication. With Message-Orient Movement, data is packaged as messages and these messages are sent when triggered by specific events. Meanwhile, Data Replication involved a data source frequently sending copies of data to the destination data warehouse, providing near-real-time updates.

Data Warehousing and the Advent of Cloud Technology

At this point in the internet’s evolution, we are experiencing a wave of new Cloud Technologies. Cloud technology is basically on-demand computer system resources that are available over the internet. Clusters of servers are integrated to provide services like data storage and computing power without the user needing to be concerned about details like which server to access or any other network details.

Benefits of Cloud Data Warehouse

Previously, if an organisation needed data warehousing capabilities then that would require, firstly, either building and configuring an on-site server or renting servers off-site and, secondly, configuring the connections between relevant assets. Either option requires significant capital outlay. Cloud-based data warehouses minimise these issues.

Cloud-based Data Warehousing services are offered at varying price points that are a fraction of what the previous options would cost in terms of capital, time and stress. Apart from ease of implementation, cloud-based data warehouse solutions also offered scalability. Previous iterations would require building capacity that took possible future growth into consideration. With cloud-based data warehouses, that question is now redundant as your package can be easily scaled to your needs, no matter how they fluctuate over time (as long as it’s within the service’s limits).

Challenges of Cloud Data Warehouse

Security is a concern for cloud-based data warehousing. This is specifically due to the fact that service providers have access to their customer’s data. While service agreements and public legislation around data privacy do exist, it must be borne in mind that it is possible that these entities could, accidentally or deliberately, alter or delete the data.

Another major security concern is the penetration of cloud systems by hackers who are constantly searching for and exploiting vulnerabilities in these systems in order to gain access to user’s personal data and data belonging to large corporations. Providers take maximum precautions in protecting user’s data. To this end, users are also offered choices in how their data is stored, such as having it encrypted in order to prevent unauthorised access.

Given the large variety of applications, businesses use today, loading all this data present in different formats into a data warehouse is a huge task for engineers. However, fully-managed data integration platform like Hevo Data (Features and 14-day free trial) help easily mitigate this problem by providing an easy, point and click platform to load data to the warehouse.

Top Five Cloud Data Warehouse Services

There are many cloud data warehouse vendors offering a wide variety of solutions. According to IT Central Station, the top 5 cloud data warehouse providers are:

Google BigQuery
Snowflake
Amazon Redshift
Microsoft Azure SQL Data Warehouse
Oracle Autonomous Data Warehouse

What are your thoughts and opinions about cloud data warehouse? Let us know in the comments.

Understanding Machine Learning Models

Arash — Thu, 01 Aug 2019 18:42:00 GMT

Machine Learning (ML) models are intended to positively impact business efficiency. By understanding how these models are created, how they function, and how they are put into production, one can fully utilize their potential to make a difference in every day scenarios.

What is a Machine Learning Model?

By creating cases within a narrow domain, such as a car insurance company assessing the risk of a particular vehicle being stolen based on known statistics, a machine learning model will use algorithms to determine probability and associate this probability with a particular outcome. While such algorithms are not necessarily limited to particular scenarios, they can be programmed to a higher degree of accuracy for specific types of questions. Below are some use cases that exemplify ideal machine learning models.

What are referred to as regression questions. These would include ‘How much’ and ‘how many’.
Classification questions that include ‘Type of object’ scenarios.
Questions that enable the model to group or cluster in order to resolve a particular scenario.
What are known as ‘abnormality detection questions’ that pinpoint unusual situations.

Engineers and data scientists use tools, frameworks and codes to build models, often from massive amounts of data.

In fact a really effective machine learning model uses enormous amounts of data that ideally has been cleaned and labelled. The process is iterative and involves both trial and error using tests and measures. Multiple steps and processes are used in creating a machine learning model. The finished model enables the computer to use different cases within a particular scenario in order to reach a viable resolution.

Using the answers to specific questions within an array of proven cases, the machine learning model provides users with guidance based on the probability that a particular solution is correct. For example, are particular symptoms indicative of a known medical problem, can this product be fixed, or is this a fraudulent financial transaction?

The Practical Utility of Machine Learning Models

Machine Learning models are intended to achieve the following outcomes:

Use on-the-fly or batch cases to integrate the model systematically
Combine several models to answer complex questions that require multi-step answers
Utilize models to assist with organizational decision making or with external contacts
Integrate workflows and processes that involve several participants
Use certain information system related algorithms with minimal code revision
Provide analytics as a service by sharing the model between multiple use cases.

Monitoring and measuring the machine learning models in a live environment is crucial. In so doing, a cycle of constant improvement is employed. While individual models are not as useful as those that are part of a more sophisticated deployment involving multiple scenarios. In such cases, the solutions suggested should be run against to a decision model that is based on a domain expert’s knowledge and consequently be implemented by using predefined business rules.

As mentioned above, a machine learning model may be designed by an insurance company using statistics that detail the likelihood of a particular car being stolen. The model will categorize a car as low, medium or high risk.

Consequently, calculating an insurance quote for a specific vehicle involves the system calling to a machine learning model which will then identify the likelihood of it being stolen. The result is then sent to the quote generation process to calculate the cost for an insurance policy.

Conclusion

Experience has shown that machine learning models need to be integrated as part of a business decision and process in order to be used effectively. These models must be able to execute requests on-the-fly and their performance within a particular knowledge domain must be monitored, measured, and improved over time.

For more information, please visit http://www.flexrule.com.

The Importance of Operational Decisions

Arash — Tue, 16 Apr 2019 07:35:00 GMT

It may sound routine, but the importance of operational decisions cannot be underestimated. After all, not a day goes by without even the smallest business making dozens, if not hundreds of operational decisions that may affect the bottom line. Elevate these to large scale companies and we are talking thousands, if not millions of actions that impact day-to-day business operations.

So what do we mean by ‘operational decisions’? The dictionary tells us that a decision is a ‘conclusion or resolution reached after consideration’. It is defined as ‘the action or process of deciding something or of resolving a question’.

From this straightforward definition, we might construe that an operational decision must have an outcome based on a procedure that is designed to resolve an everyday or, in most cases, an ‘unexceptional’ occurrence. This can be called the ‘decision logic’.

It is important to point out, however, that there are differences between operational decisions and those that may be viewed as strategic to the business. Operational decisions are often highly structured, repetitive and routine. In other words, you can model them once and then reuse them ad infinitum for hundreds or thousands of every day transactions. For example, an operational decision may consider compliance with state regulations, or the possibility of a fraudulent transaction, or a calculation of taxation, or an exception to a claims process.

This is precisely why operational decisions are very often excellent candidates for automation.

To demonstrate, an operational decision structure typically looks something like this:

Operational Decision structure. It includes the “decision logic”, input data and the conclusion.

The judgement or action to be taken in each case could involve a list, a simple value, a calculation, a configuration, a price, a guideline, and several other routine outcomes. For example, it could be a list of available options for treatment of a patient, or the best prices for upselling and product bundling, or a billing calculation for a customer, or specific procedures and guidelines, and so on.

The bottom line is that all operational decisions require an input in order to describe parameters, situations and cases. The decision logic is then executed and a result is produced.

The critical piece here is a combination of effectiveness and efficiency, especially when businesses today are challenged by rapid change and competitive pressure. Yet it is a fact that in many cases, the actual logic behind an operational decision is not at all transparent to the organization. That may be because it is buried in legacy processes and applications or embedded in the minds of subject matter experts who will inevitably move on to other pursuits in a matter of time.

The Importance of Automating Operational Decisions

All too often companies use traditional approaches in an attempt to harness operational decisions. This might involve building applications using code and low-code platforms, iBPMS and other technologies. The problem, of course, is that the operational decision becomes ‘set in stone’, as it is buried in some hard-coded application that is difficult to change. As an interim solution when change request delays occur, some may try to compensate by resorting to Excel and other spreadsheet solutions. Other business-oriented solutions like Business Rules Management Systems (BRMS), Business Process Management (BPM) and Decision Management Systems (DMS) may also come into play as companies seek to memorialise their operational decision-making processes.

The problem with these types of traditional approaches is that when these are challenged by the necessary frequency of high volume changes inherent in operational decision-making, they must be managed independently. This in turn means that decision management becomes a maintenance nightmare. In short, the organisation simply cannot keep pace with the rate of essential change to operational decisions. It is simply not enough to change one line of code or one business rule here and there while other necessary changes stay in queue until such time as someone has time to explore the deep, dark recesses of traditional systems and figure out what to do. This approach inevitably leads to less effective results.

Obviously, there is an essential need to balance the control between IT and the business by using a solution that meets the need for effective and efficient operational decision-making. That’s where FlexRule comes in. Read more about our ground-breaking solutions at www.flexrule.com.

The Case Against Process Mapping and Why You Should Do Process Discovery Instead

Simona from Minit — Mon, 06 Aug 2018 08:44:00 GMT

Process mapping vs. process discovery is akin to perceived reality vs. reality; the former rooted in subjectivity, the latter rooted in verifiable data. Elements of process mapping creep into process discovery, restricting an absolute dichotomy between the two. However, the critical differentiator between mapping and discovery lies in the distinction between fact and fiction. Process discovery is primarily concerned with concrete, verifiable data (event logs, digital footprints), while process mapping relies on subjective first-hand remembered events.

Let us first review the core definitions of process mapping and process discovery before moving to the case against process mapping in favor of process discovery.

Key Differences Between Process Mappingig and Process Discovery

Process mapping is the human-side of establishing an ‘as-is-process.’ It’s concerned with measuring and comparing a defined objective against an organization's larger vision to ensure process are aligned with a company’s core competencies, capabilities, and overarching values. While the end game aims for process improvement, a significant element of subjectivity and unintentional validation techniques are laced within manual process mapping. Perceived reality is flawed regardless of intentions.

Automated process discovery, on the other hand, is exclusively concerned with verifiable data logs, providing an accurate picture of how processes are performed, rather than the idealized model of how they should be performed, or how employees think they are performed. Additionally, process discovery takes the white space between information systems and seemingly unrelated events and builds bridges with data rather than assumptions. Anomalies and outliers are accurately weighted without being unfairly amplified or ignored

Process Mapping	Process Discovery
Manual	Automated
Subjective	Objective
Remembered events	Verifiable event logs
Human validation	Data-driven
Limited scalability	Full scalability
Process details must be known by employees	No employee knowledge of process details needed
Slow, drawn-out	Fast, continuous
Too much or too little detail	Exact reality
White space between IT systems and processes unknown	White space between IT systems and processes bridged
Outliers ignored	Outliers appropriately weighted

Process Discovery vs Process Mapping — Why Discovery is Best

The case for process discovery is strong, particularly when midsize and enterprise level companies are looking to initiate Business Process Improvements (BPI). From knowledge gaps bridged to socio-cultural subjectivity eliminated, here are the top reasons to choose process discovery over process mapping.

1. You don’t know what you don’t know (white space)

White space is the unseen area between various systems, departments, and functions at the edges of a process. One of the most significant challenges in manual process mapping is effectively extracting information from employees involved in a process. Piecing together “remembered activities” to create a process map will inevitably be riddled with knowledge gaps, employee validation, and human subjectivity. In other words, you don’t know what you don’t know, and “unseen” events will not be included in the map.

Process discovery captures all the nuances of a process, including statistical information, process exceptions, unusual transactions, deviations, potential process risks, bottlenecks, and variants. Process discovery bridges the gaps between the individual process steps across multiple ERP and IT systems. Automated process discovery delivers a detailed process map rich with data and flexible for interactive analysis.

2. Map unstructured data from process unknown to employees

Concerning white space and knowledge gaps, some processes, and steps in a process are entirely unknown to employees and, therefore, must rely on digital event logs to create a picture of a process. Consider a global supply chain operating on Just-In-Time manufacturing principles. This fast-moving, lean approach to production relies on multiple actions triggered by a single event. Building on verifiable, time-stamped data logs from hundreds to hundreds of thousands of tiny events is something humans simply cannot map. Process discovery systems, like Minit, take this unstructured data and automatically create a process model to enable in-depth analysis. Minit technology is able to take various data inputs from scratch and deliver pattern recognition. This enables companies to discover processes without prior process knowledge or specifying an existing model.

3. Eliminate socio-cultural behavior from analysis

Humans make decisions based on emotions. It’s a fact wired into our neurological pathways. This doesn’t mean we are illogical, far from it. It means we gather information, process this information from our lens of reality, and then use the frontal lobe of our brain to make a decision — the area responsible for emotional expression, problem-solving, memory, language, and judgment.

Without intent for sabotage, humans will deliver a subjective view of reality, as well as use unintentional validation techniques. It’s not lying; it’s how we communicate as humans. Automated process discovery reads between the lines of data logs, not words, eliminating the presence of socio-cultural behavior from a process analysis. Additionally, what manual process mapping may express as statistical noise, process discovery can appropriately highlight as inefficiencies in business processes.

4. Focus on accuracy and speed

Have you ever seen a whiteboard overloaded with sticky notes — diamonds, arrows, mini-sticky notes upon larger sticky notes? An array of orange, pink, blue and green activities? This is process mapping. Process mapping can, and should, be accompanied by technology like Microsoft Visio and purpose-built process mapping tools, but keep in mind that these tools help you take sticky notes off a white board and build sticky notes on a computer screen. Your big win here is that a humid day won’t wipe progress off the board. This approach is still relying on human inputs, not data inputs, bringing accuracy level to a low.

In terms of speed, calculate the total human hours needed for individual staff interviews, facilitated discovery workshops, analysis of existing documentation and direct work observation. Then compare this to process mining software that plugs in, transforms unstructured data into meaningful maps, and delivers a flexible, in-depth process analysis based on hard data. This is an essential part of understanding the real cost of going through a BPI transformation.

5. Unlimited scalability

Last but not least, a massive benefit of automated process discovery over manual process mapping is scalability. Once systems are connected and process mining software established, a process can be endlessly reanalyzed with little to no additional effort. As subtle changes in the process are made during the discovery period, technology will capture this immediately and include it in data mapping. Businesses need to optimize business processes continuously, and scalability is a big part of making this financially feasible.

See Process Discovery Technology in Action

Minit software analyzes concrete data from various systems to discover how business processes flow in reality. Our platform automatically recreates process maps from traces of user actions (electronic footprints) left within applications. It provides an accurate picture of how users — employees, suppliers, customers, etc. — are performing their duties or actions rather than the idealized model of what they are supposed to be doing.

We’d love to show you the power of Minit and how it can help deliver an effective BPI transformation at your organization. Get in touch with our team to learn more and request a trial of Minit.

What is Predictive Analytics and Why Does It Matter?

Arash — Tue, 13 Mar 2018 07:27:00 GMT

What is Data Analytics?

To understand the meaning of Predictive Analytics, let’s describe what Data is first. Data is a collection of facts, information, and observations related to a context. The data can be either structured or unstructured, stored in databases, spreadsheets, files, etc. Data analytics is the science of examining the data to drive conclusions and find answers to particular questions.

Data Analytics can be defined and applied at different levels:

As the above Gartner diagram shows, we can define four levels of analytics:

Descriptive tells us “What happened?” which is pretty much all of the standard reporting capabilities that we have seen so far in any computerized system. For example, a sales report generated for a customer to understand the transactions for the current year.
Diagnostic tells us “Why did it happen?” by using advanced techniques such as drill-down, data discovery, data mining and correlations (e.g., BI finds the relationship between data points and helps us to understand Why a specific event has occurred).
Predictive tells us “What will happen?” by using historical data and an understanding of the past in order to predict the future. This is called supervised learning in AI.
Prescriptive tells us “What should I do?” based on the information that we could predict using “Predictive” analytics. The system can prescribe the best next action, offer, decision, etc., to the user or it can fully automate the cycle (when possible)

Let’s see what is so special about Predictive Analytics and how it can help your business.

What is Predictive Analytics? Why does it matter?

Data Analytics nowadays is a hot topic, especially with the advancement of AI and accessibility of fast and cheap computing power. Predictive Analytics is a special branch of AI that uses supervised learning, statistical techniques from predictive modelling, machine learning, and data mining to analyze current and historical data in order to make predictions about the future. Predictive Analytics can play a big role in helping you to win the competition.

How? Predictive Analytics can harness the data and create a unique opportunity for organizations.

Real Life Examples

Below are just a few scenarios in which you can use Predictive Analytics to find a new opportunity and take action on it:

Forecasting sales volume based on last year’s sales history of a product and current orders to ensure you will have always have the right amount of the item in stock. Potentially, you can use a Workflow to automate the whole approval-buy cycle of the purchase.
Finding the probability of a patient illness based on the history of similar patients in the medical center during the current season, and helping the GP to identify and diagnose illnesses based on the probability of these hypotheses. Potentially, you can use Decision Automation to prescribe the right set of activities, drugs, and regimens for the patients.
Classifying customers based on buying habits, and sending the right offer to the right customer to boost sales. Potentially, you can automate the entire cycle and let the system send out vouchers, sales offers, etc., using the right groups of products to the correct sets of customers. Then the system monitors the customers behavior again and feeds the results back to the model. This improves the quality of the next offer in order to maximize profitability.

Conclusion

It is important to understand that the value of Predictive Analytics has become a reality. However, it will have positive impact on our day-to-day jobs only if we utilize them and close the loop of the Observe-Orient-Decide-Act (OODA), not just show these on a beautiful dashboard.

These are just few examples and in the next post I will show you how to build a predictive model using FlexRule.

Patient diagnosis using Predictive Analytics

Goli Tajadod — Mon, 12 Mar 2018 07:59:00 GMT

Patient diagnosis using Predictive Analytics

In previous post we discussed different levels of Analytics, here we show a practical example of Predictive Analytics. What if Doctors and Patients could not just get a second opinion, but a third and fourth, and fifth, and so on? We believe that our doctors are all medical experts but patient diagnosis using Predictive Analytics can help them to make more informed decisions based on current and historical data.

“Misdiagnosis accounts for about one-third of all medical error. Autopsy studies show that doctors seriously misdiagnose fatal illnesses about 20 percent of the time. “If you look at settled malpractice claims,” Britto said, “diagnosis error is about twice or three times as common as prescription error”.

Ian Ayres; Super Crunchers, How anything can be predicted, page 97

As explained by Ian Ayres, the Doctor’s diagnosis phase by doctors is the most important step in defining the patient’s journey based on the individual patient’s symptoms. When Predictive Analytics are involved in this decision, then additional historical data of similar patient’s symptoms would help the doctors to make more insightful decisions.
In the latest version of FlexRule, we added a new sample project called patient diagnosis using Predictive Analytics. This project demonstrates how a patient’s history at the same medical centre (city area) in a particular time frame would help doctors to make an informed decision for a new patient with a similar list of current symptoms.

The model has three main steps: read historical data, use the Naive Bayes (NB) algorithm to train the model, and, of course, predict the diagnosis.

Reading Patients History

The patient observation dataset has been read from a DataSheet (.CSV file). The dataset contains 20 patients with different symptoms and final diagnoses confirmed by their doctors and through lab tests. For example, the first row, which is patient one, had symptoms like sneezing and a sore throat and therefore was diagnosed with cold. The second row, which is patient two, had symptoms like fatigue, stuffy nose and sneezing, which was also found to be a cold.

Building a Trained Model

In the second step, we train (or reload an existing) predictive model using the Naive Bayes algorithm (NB), which is one of the scalable classification algorithms for these types of dataset. This algorithm has been successfully applied to many medical problems with large amounts of data and features. Now our model is ready to predict the diagnosis for the next patient based on his current symptoms.

Predicting a New Situation

In this step we pass the data of our new patient’s current symptoms to the predictive model. Patient 21 has not been diagnosed by a doctor yet and he has stuffy nose, sneezing, and sore throat:

{
    "Fatigue": null,
    "Stuffy Nose": "T",
    "Sneezing": "T",
    "Sore Throat": "T"
}

Based on historical data and the current patient’s symptoms, our predictive model calculates the percentage probability of each disease as shown below:

Conclusion

As the result clearly shows, patient 21 is likely to have a cold with a probability of 74.81%.
The probability of a flu and allergies are 8.76% and 16.41% respectively, which are much lower than the probability of the patient having a cold.
This is a simple example, but it shows how FlexRule and predictive modelling can help a doctor make better quality patient diagnosis.

5 Business Problems You Can Solve Using SQL Temporal Tables

Bert Wagner — Tue, 11 Jul 2017 11:08:00 GMT

It’s 4:30 pm on Friday and Mr. Manager comes along to tell you that he needs you to run some important ad-hoc analysis for him.

Previously this meant having to stay late at the office, writing cumbersome queries to extract business information from transactional data.

Lucky for you, you’ve recently started using Temporal Tables in SQL Server ensuring that you’ll be able to answer your boss’s questions and still make it to happy hour for $3 margaritas.

Sound like a plan? Keep reading below!

The Data

For these demos, we’ll be using my imaginary car rental business data. It consists of our temporal table dbo.CarInventory and our history table dbo.CarInventoryHistory:

I’ve upgraded my business — we now have FOUR Chevy Malibus available for you to rent

Business Problem #1 — “Get me current inventory!”

To get our current inventory of rental cars, all we have to do is query the temporal table:

SELECT * FROM dbo.CarInventory

That’s it.

I know this query seems lame — it’s just a SELECT FROM statement. There are no FOR SYSTEM TIME clauses, WHERE statements, and no other interesting T-SQL features.

But that’s the point! Have you ever had to get the “current” rows out of a table that is keeping track of all transactions? I’m sure it involved some GROUP BY statements, some window functions, and more than a few cups of coffee.

Temporal tables automatically manage your transaction history, providing the most current records in one table (dbo.CarInventory) and all of the historical transactions in another (dbo.CarInventoryHistory). No need for complicated queries.

Business Problem #2 — “How many miles on average do our customers drive each of our cars?”

In this example, we use FOR SYSTEM_TIME ALL and a plain old GROUP BY to get the data we need:

SELECT
 CarId, AVG(Mileage) AS AverageMileage
FROM
 dbo.CarInventory FOR SYSTEM_TIME ALL
WHERE
 InLot = 1 -- The car has been successfully returned to our lot
 AND SysStartTime > '2017-05-13 08:00:00.0000000' -- Ignore our initial car purchase
GROUP BY
 CarId

Some cars get driven a lot more. Causation or correlation?

FOR SYSTEM_TIME ALL returns all rows from both the temporal and history table. It’s equivalent to:

SELECT * FROM dbo.CarInventory 
UNION ALL 
SELECT * FROM dbo.CarInventoryHistory

Once again, there isn’t anything too fancy going on here — but that’s the point. With temporal tables, your data is organized to make analysis easier.

Business Problem #3 — “How many cars do we rent out week over week?”

Here at Wagner Car Rentals we want to figure out how often our cars are being rented and see how those numbers change from week to week.

SELECT
 CurrentWeek.CarId,
 CurrentWeek.RentalCount AS CurrentRentalCount,
 PreviousWeek.RentalCount AS PreviousRentalCount
FROM
 (
 SELECT
 CarId,
 COUNT(*) AS RentalCount
 FROM
 dbo.CarInventory FOR SYSTEM_TIME FROM '2017-06-05' TO '2017-06-12'
 WHERE
 InLot = 0 -- Car is out with the customer
 GROUP BY
 CarId
 ) CurrentWeek
 FULL JOIN
 (
 SELECT
 CarId,
 COUNT(*) AS RentalCount
 FROM
 dbo.CarInventory FOR SYSTEM_TIME FROM '2017-05-29' TO '2017-06-05'
 WHERE
 InLot = 0 -- Car is out with the customer
 GROUP BY
 CarId
 ) PreviousWeek
 ON CurrentWeek.CarId = PreviousWeek.CarId

In this query, we are using FOR SYSTEM_TIME FOR/TO on our temporal table to specify what data we want in our “CurrentWeek” and “PreviousWeek” subqueries.

FOR/TO returns any records that were active during the specified range(BETWEEN/AND does the same thing, but its upper bound datetime2 value is inclusive instead of exclusive).

Business Problem #4 — “What color cars are rented most often?”

We’re thinking of expanding our fleet of rental vehicles and want to purchase cars in the most popular colors so we can keep customers happy (and get more of their business!). How can we tell which color cars get rented most often?

SELECT 
 CarId, 
 Color, 
 COUNT(*)/2 AS RentalCount -- Divide by 2 because transactions are double counted (rental and return dates)
FROM 
 dbo.CarInventory FOR SYSTEM_TIME CONTAINED IN ('2017-05-15','2017-06-15')
GROUP BY 
 CarId,
 Color

Here we use CONTAINED IN because we want to get precise counts of how many cars were rented and returned in a specific date range (if a car wasn’t returned — stolen, wrecked and totaled, etc… — we don’t want to purchase more of those colors in the future).

Business Problem #5 — “Jerry broke it. FIX IT!”

The computer systems that we use at Wagner Car Rentals are a little…dated.

Instead of scanning a bar code to return a car back into our system, our employees need to manually type in the car details. The problem here is that some employees (like Jerry) can’t type, and often makes typos:

SELECT * FROM dbo.CarInventory FOR SYSTEM_TIME ALL WHERE CarId = 4

Having inconsistent data makes our reporting much more difficult, but fortunately since we have our temporal table tracking row-level history, we can easily correct Jerry’s typos by pulling the correct values from a previous record:

;WITH InventoryHistory 
AS 
( 
 SELECT ROW_NUMBER () OVER (PARTITION BY CarId ORDER BY SysStartTime DESC) AS RN, * 
 FROM dbo.CarInventory FOR SYSTEM_TIME ALL WHERE CarId = 4 
) 
--SELECT * FROM InventoryHistory
/*Update current row by using N-th row version from history (default is 1 - i.e. last version)*/ 
UPDATE dbo.CarInventory 
 SET Color = h.Color 
 FROM 
 dbo.CarInventory i 
 INNER JOIN InventoryHistory h 
 ON i.CarId = h.CarId 
 AND RN = 2

Typos fixed!

Although we could have fixed this issue without using a temporal table, it shows how having all of the row-level transaction history makes it possible to repair incorrect data in more difficult scenarios. For even hairier situations, you can even roll-back your temporal table data.

Conclusion

Temporal tables are easy to setup and make writing analytical queries a cinch.

Hopefully writing queries against temporal tables will prevent you from having to stay late in the office the next time your manager asks you to run some ad-hoc analysis.

-----

Bert Wagner

https://blog.bertwagner.com

Modeling your way to a great backlog

Candase Hokanson — Tue, 31 May 2016 12:24:00 GMT

If you’re a business analyst whose company recently made the move to agile, you may be wondering where you fit in when there is no business analyst role. Or maybe you made the move to be an agile business analyst or product owner but don’t know how your traditional business analyst skills figure into this new agile world. Well the good news is that even in agile frameworks with no official business analyst role, business analysis still needs to be performed for every product so that we know we’re building the right thing. It may be called something else and we may do it differently, but it is still a vital part of the agile process.

With this post, I want to talk about one piece of business analysis - visual modeling - and how it fits into agile and building a great backlog. In waterfall projects, the business analyst will typically gather all the requirements up front (prior to design and development) utilizing visual models to enhance and give context to her requirements. In agile, we do everything with “just enough” detail, “just in time” for the development team to start working on a given requirement or user story, so many people have taken the route that the user story along with conversation is enough and we don’t need visual models anymore. This couldn’t be more wrong! Especially with so many large, global companies moving to agile and having distributed team with lots of dependencies, visual models can help bridge communication gaps that co-located agile teams can cover with conversation.

So, why visual models in agile? Visual models arguably are even more important in agile projects than in waterfall projects because there is a low cost to build or change them. They are easy tools to use to gain understanding between the business stakeholders and the development team without having to spend a lot of time writing requirements or user stories. Additionally, visual models allow the product owner or business analyst to view the whole product and vision while focusing on a sub-set for delivery - enabling the product owner to deliver the most value the soonest. Visual models are a great supplement to the backlog and user stories to gain a richer understanding of the product and the needs of the users. They also help the product owner or business analyst find missing stories and acceptance criteria.

Which models are best in an agile process? We have 22 visual models in RML®, but not all of those are useful all the time, especially on an agile project. However, I’ve found that there are about 5 models that are used on almost every agile project I’ve worked on. Let’s walk through each one - in this blog post, I’m just going to give an overview of each visual model, but you can find more information in the links below.

Business Objectives Model: The business objectives model tells the team why they are working on a project or building a product. It is similar to a product vision, but gives concrete objectives that we want to solve with the project/product. On agile projects, this is probably the most important model because the product owner or business analyst should be tracing every single user story back to the business objectives model to ensure that we are only building what is needed and useful to the user.

Process Flow: Process flows detail out a business process from the user’s perspective. They can be at multiple levels (typically L1, L2 and L3 - starting at the highest, most-abstract level and working down in detail) and so are great for agile projects! At the beginning of a project, the product owner or business analyst can define the L1, high-level process to identify epics or features, and as iterations progress, can dig deeper into L2 and L3 detailed process flows. Each step in a lower level process flow is likely to be one or more user stories.

Feature Tree: A feature tree is a visual model that lists out all features for a product/project in a hierarchical tree format. In agile, this is a good tool to keep up with requested features because it is easy to update. By color coding or making important features closer to the “trunk” of the feature tree, you can easily show iterations or releases.

Business Data Diagram: The business data diagram shows all data objects in a project or product and how they relate to each other. On agile projects, this serves two purposes: as a catalog of the overall data needs (hard to show in a user story) for use in building databases and as a source of acceptance criteria to enforce the relationships between data objects. This is a lower level model and may be started in a sprint 0, but would need to be kept updated in subsequent sprints.

Decision Tree or Decision Table: Finally, decision trees and decision tables detail out system logic and how the system should respond to various input decisions. These can be used to expand a given user story with decision logic or can be used as the acceptance criteria themselves. Since one of the most common ways of writing acceptance criteria is the “Given, When, Then” format, this lines up nicely to decision tables, where the “given” is a precondition, the “when” is a trigger or decision and the “then” is the outcome. Decision tables ensure that every combination of decisions and outcomes is considered, so when using these for acceptance criteria it is clear to the developers and testers what should happen in every instance.

These are the visual models I’ve found useful on my agile projects, which ones do you use?

Manage The Surge In Unstructured Data

XOCom — Fri, 09 Oct 2015 12:24:00 GMT

The average enterprise will need to manage 50 times more information by 2020 and while the amount of IT staff will only increase by 1.5%. This surge in unstructured data creates many tough challenges for business.

By XO Communications

RML® Model 4 – Data Dictionary

Seilevel — Thu, 05 Aug 2010 18:09:00 GMT

When creating a data dictionary, it is critical not to design with it. You should not be attempting to create a database nor should you be showing the relationships of data. This will be a detailed description of the data involved in your project. It will also be one of the largest morale draining approval processes you can have for a document. They will be very time consuming an have the potential to have a good deal of churn over the data items they hold.

One of my former projects demanded a great amount of detail be gathered from the existing systems to be implemented in a new outside system. Unfortunately there were several large problems with the way Data Dictionaries were used. One of the two largest problems was that there were upwards of twenty different data dictionaries that were being worked on independently by different functional areas of the project. The other huge issue was that only fields that were visible to the business were being captured.

Having multiple data dictionaries for the same project causes many issues and can lead to delayed development and incomplete requirements. If there are 10 different Data Dictionaries that all call for a field identified as ‘Credit’, how does one figure out if they are all the same field? If they are the same field, which of the different entries has the correct business rules and attributes? If they are not the same field, how will the names be reconciled? If they are different, which one was being referred to in other documentation and models? And what about the one field labeled ‘CustCred’. Was that the same? Were all the business SME’s aware that ‘CustCred’ was included in only one of the data dictionaries?

Failure to engage the Business and IT in creating a Data Dictionary can lead to a plethora of data not being identified and documented. Unfortunately when only the business is involved in creating a Data Dictionary, it tends to only get populated with fields that can be seen, either on the screen or in reports. IT will be able to provide input around these fields. What is not being seen is that when you enter your account number and password, the system is keeping a log of your user id, the date the login attempt occurred, the outcome, and your IP address. Without IT there, you could have just lost 4 of the 6 fields required to login. Additionally, if you have to go back a separate time with the IT group, you will have to bring the Business in even more to make sure that the business name you have recorded is the same as the field IT identified. Just because it says customer name on the screen doesn’t eliminate the possibility that it is actually ‘cust_legal_name’ in the database.

I have found the following fields to be a robust baseline for any data dictionary I create:

-ID – unique ID to keep track of the data fields

-Category – Organizational field that can help group data

-Business Name – Official or most widely recognized name for the data

-Aliases – Abbreviations or alternative names the data could go by

-Data Field – Field used in the database (mostly used on gap projects)

-Description – Short description of the field for context

-Formula – If the field is calculated, include the formula used to come up with the value here

-Business Rules – Any actions or restrictions that must apply to this data

-Data Type – generic data type such as ‘alpha’, ‘numeric’, ‘alphanumeric’, ‘currency’, ‘date’, or ‘time’

- Length – Maximum allowed length of the field

-Owner – Individual or group who currently owns an action item for the field

-Status – Draft/Reviewed/Approved/Removed/etc.

- Notes – Misc notes to be taken about the field such as links or individual’s comments

-Issues – List of unresolved issues or action items with the field

Keep in mind that the Data Dictionary is best in situations where you have an undefined data model or in situation where you are involving new systems. This data model will become less useful when dealing with a smaller or more defined/understood data set or if you are short on time.

Unfortunately this model is also not a visual model. This means that it will be much harder for people consuming it to quickly learn the data being presented and it will also be difficult to quickly interpret the data being presented.

You are almost always going to find this document living as an excel spreadsheet. If you plan on having multiple large groups working on the same project, you should consider having a collaboration tool for this document such as Google Docs or SharePoint lists. Multiple lists being worked on separately can cause massive headaches and prove to be massive time sink when trying to correct and compile. As a best practice, one should never delete data either, but rather mark it as removed or do a strikethrough formatting. This will allow you to know that it was documented but deemed unnecessary instead of forgetting about that one remote field and researching it again to find out you cut it already.

Pro Tip for excel users:

Holding down Alt and pressing Enter (Alt + Enter) while typing in a cell will allow you to return

to the next line within the cell as if it were a word document. This will make it easier to format notes or add multiple aliases/data fields to a single row.

Fun tip- for those who have sympathy for making others do a data dictionary review: bring koosh balls, stress relievers or fun packs of play-doh for your attendees to keep their brains semi-active during the reviews.

by jheep

Want more models? check us out here.

How to write Entity Relationship Diagrams

Craig Brown — Fri, 29 Aug 2008 04:29:00 GMT

E R Diagram

View SlideShare presentation or Upload your own. (tags: er diagram)